Collaborators

Claire McGrath Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region,

Kevin See Biometrician, Biomark Inc, Boise, ID,

Goals

Salmon redd counts are widespread method to estimate the number of returning adult spawners. However, despite its prevalence in the Northwest, the reliability of redd counts is unknown. This work is focused on developing a statistical model to estimate the observer error in redd surveys, using a variety of covariates related to the habitat and the observer. We described three types of observer error:

  • omission rate, \(\omega\) (proprotion of redds available to be counted that were missed by the observer)
  • commission rate, \(\eta\) (rate of redds counted by an observer that were not actually redds)
  • net error, \(\gamma\) (ratio of observed redds to true redds). This was modeled using log(net error) as the reponse.

Methods

Possible covariates in each error model are shown in Table 1. To make comparisons with AICc, the random effects must be identical across all models. Therefore, we ensured that the random effect of year was added to any model that didn’t have it.

Table 1: Possible covariates included in each observer error model.

Type Covariate Air Ground
Random Reach X X
Random Surveyor X
Random Year X X
Fixed AveBadCond X
Fixed AveCanopy X X
Fixed AveDepth X X
Fixed AveSunny X
Fixed AveWidth X X
Fixed EscapeEst X X
Fixed Experience3 X
Fixed I(AveDepth^2) X X
Fixed LYabund X X
Fixed LYabund:PeakQ X X
Fixed PeakQ X X
Fixed redd_dens_obs X X
Fixed Slope X X

All covariates were z-scored, and all models were fit using the glmer or lmer functions from the lme4 package (Bates et al. 2015) in R software (R Core Team 2019). The amount of variation explained by fixed and random effects was calculated using the methods of Nakagawa and Schielzeth (2013). Using estimated predictions of the rates for omission (\(\hat{\omega}\)), commission (\(\hat{\eta}\)) and net error (\(\hat{\gamma}\)), we predicted the number of actual redds by either dividing the observed counts, \(c\), by estimates of net error, or by multiplying the observed counts by 1 - estimated rate of commission, and then dividing by 1 - estimated rate of omission.

We performed a cross validation by dividing each survey type data into 5 training datasets where 20% of the data was withheld for testing, and then fitting the naive and best AICc model formulations to the remaining data, and then using those fits to predict the error rates and true number of redds for each survey in the year that had been withheld.

\[ \begin{aligned} redds_{ne} &= \frac{c}{\hat{\gamma}} \\ redds_{om} &= c * \frac{1 - \hat{\eta}}{1 - \hat{\omega}} \end{aligned} \]

The observed error rates are showin in Figure 1.

Figure 1: Observed error rates.

Figure 1: Observed error rates.

Results

Model Coefficients

The model coefficients of the full, best (by AICc) and model averaged models are shown in Table 2.

Table 2: Estimated coefficients for various observer error models.

Survey Resp Covariate avg best full
Ground Com (Intercept) -1.157 -1.157 -1.157
Ground Com AveCanopy 0.007 0.007 0.007
Ground Com AveDepth 0.128 0.128 0.128
Ground Com AveWidth -0.141 -0.141 -0.141
Ground Com EscapeEst -0.311 -0.311 -0.311
Ground Com Experience3.L -0.234 -0.234 -0.234
Ground Com Experience3.Q 0.275 0.275 0.275
Ground Com I(AveDepth^2) 0.056 0.056 0.056
Ground Com LYabund -0.222 -0.222 -0.222
Ground Com LYabund:PeakQ -0.735 -0.735 -0.735
Ground Com PeakQ -0.681 -0.681 -0.681
Ground Com redd_dens_obs 0.438 0.438 0.438
Ground Com Slope 0.171 0.171 0.171
Ground Net (Intercept) -0.257 -0.249 -0.304
Ground Net AveCanopy -0.041 - -0.026
Ground Net AveDepth 0.079 - 0.057
Ground Net AveWidth -0.094 - -0.075
Ground Net EscapeEst -0.077 - -0.077
Ground Net Experience3.L 0.318 - 0.295
Ground Net Experience3.Q -0.169 - -0.180
Ground Net I(AveDepth^2) 0.009 - 0.005
Ground Net LYabund 0.092 - -0.031
Ground Net LYabund:PeakQ 0.040 - -0.168
Ground Net PeakQ -0.031 - -0.159
Ground Net redd_dens_obs 0.143 0.143 0.179
Ground Net Slope -0.042 - 0.003
Ground Omi (Intercept) -0.265 -0.265 -0.265
Ground Omi AveCanopy 0.018 0.018 0.018
Ground Omi AveDepth 0.109 0.109 0.109
Ground Omi AveWidth -0.107 -0.107 -0.107
Ground Omi EscapeEst -0.014 -0.014 -0.014
Ground Omi Experience3.L -0.606 -0.606 -0.606
Ground Omi Experience3.Q 0.613 0.613 0.613
Ground Omi I(AveDepth^2) -0.055 -0.055 -0.055
Ground Omi LYabund -0.151 -0.151 -0.151
Ground Omi LYabund:PeakQ -0.278 -0.278 -0.278
Ground Omi PeakQ -0.005 -0.005 -0.005
Ground Omi redd_dens_obs -0.186 -0.186 -0.186
Ground Omi Slope 0.301 0.301 0.301
Survey Resp Covariate avg best full
Air Com (Intercept) -1.331 -1.327 -1.383
Air Com AveBadCond 0.003 - 0.008
Air Com AveCanopy -0.152 - -0.232
Air Com AveDepth -0.043 - -0.097
Air Com AveSunny 0.345 0.345 -
Air Com AveWidth 0.135 - 0.075
Air Com EscapeEst -0.313 - -0.313
Air Com I(AveDepth^2) 0.013 - 0.001
Air Com LYabund 0.442 - 0.344
Air Com LYabund:PeakQ 0.314 - 0.032
Air Com PeakQ -0.324 - -0.526
Air Com redd_dens_obs 0.070 - 0.076
Air Com Slope 0.021 - 0.177
Air Net (Intercept) -0.190 -0.183 -0.241
Air Net AveBadCond 0.003 - 0.011
Air Net AveCanopy -0.148 - -0.159
Air Net AveDepth 0.094 - 0.056
Air Net AveSunny 0.220 0.220 -
Air Net AveWidth 0.025 - -0.023
Air Net EscapeEst -0.106 - -0.106
Air Net I(AveDepth^2) 0.013 - -0.002
Air Net LYabund 0.134 - 0.059
Air Net LYabund:PeakQ 0.158 - 0.000
Air Net PeakQ -0.084 - -0.173
Air Net redd_dens_obs 0.109 - 0.115
Air Net Slope -0.069 - 0.050
Air Omi (Intercept) -0.528 -0.528 -0.430
Air Omi AveBadCond 0.004 - 0.004
Air Omi AveCanopy 0.120 - 0.120
Air Omi AveDepth 0.064 - 0.064
Air Omi AveSunny -0.257 - -
Air Omi AveWidth -0.178 - -0.178
Air Omi EscapeEst 0.138 - 0.138
Air Omi I(AveDepth^2) -0.042 - -0.042
Air Omi LYabund 0.176 - 0.176
Air Omi LYabund:PeakQ 0.123 - 0.123
Air Omi PeakQ 0.247 - 0.247
Air Omi redd_dens_obs -0.546 -0.546 -0.529
Air Omi Slope 0.078 - 0.078

Ground Surveys

The relative importance of each covariate in each model is shown in Figure 2, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 3. Observed versus predicted rate plots are shown in Figures 4, 6 and 8.

Figure 2: Relative importance of each covariate in ground-based observer error models

Figure 2: Relative importance of each covariate in ground-based observer error models

Figure 3: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Figure 3: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Omission

Figure 4: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 4: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 5: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Figure 5: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Commission

Figure 6: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 6: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 7: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Figure 7: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Net Error

Figure 8: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 8: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 9: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Figure 9: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Leave-One-Out Cross Validation

Rate Estimates

We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects).

Redd Estimates

For ground-based surveys, both methods provided fairly unbiased estimates of the true number of redds (Figure 10), although the omission/commision models had slightly higher absolute and relative bias (Table 3).

Figure 10: Boxplots of absolute and relative bias for each type of predictive model.

Figure 10: Boxplots of absolute and relative bias for each type of predictive model.

Table 3: Summary statistics of predictions of total redds from leave-one-out cross validation using the net error and the omission/commission models.

Model Median # Obs. Redds Median # True Redds Median Adjustment Median Abs. Bias Median Rel. Bias (%) RMSE
Best Net Error 36 38 3.8 0.2 0.6 18.0
Best Omis / Comm Error 36 38 4.8 0.6 1.3 15.3
Naive Net Error 36 38 2.9 0.1 0.4 18.7
Naive Omis / Comm Error 36 38 4.0 0.3 1.0 17.5
Observed 36 38 - -2.0 -8.0 18.9
Figure 11: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Figure 11: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Air Surveys

The relative importance of each covariate in each model is shown in Figure 12, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 13. Observed versus predicted rate plots are shown in Figures 14, 16 and 18.

Figure 12: Relative importance of each covariate in ground-based observer error models

Figure 12: Relative importance of each covariate in ground-based observer error models

Figure 13: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Figure 13: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Omission

Figure 14: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 14: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 15: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Figure 15: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Commission

Figure 16: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 16: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 17: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Figure 17: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Net Error

Figure 18: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 18: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 19: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Figure 19: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Leave-One-Out Cross Validation

Rate Estimates

We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects).

Redd Estimates

For air-based surveys, both methods provided estimates of the true number of redds that were biased high (Figure 20). However, the net error models had lower absolute and relative bias, as well as a smaller root squared mean error (RMSE) (Table 4).

Figure 20: Boxplots of absolute and relative bias for each type of predictive model.

Figure 20: Boxplots of absolute and relative bias for each type of predictive model.

Table 4: Summary statistics of predictions of total redds from leave-one-out cross validation using the net error and the omission/commission models.

Model Median # Obs. Redds Median # True Redds Median Adjustment Median Abs. Bias Median Rel. Bias (%) RMSE
Best Net Error 32 38 6.1 0.6 6.5 24.3
Best Omis / Comm Error 32 38 7.7 0.2 2.1 22.8
Naive Net Error 32 38 5.3 -0.7 -2.2 27.5
Naive Omis / Comm Error 32 38 6.8 -0.9 -3.0 25.5
Observed 32 38 - -6.0 -18.2 28.7
Figure 21: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Figure 21: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Discussion

For both ground- and air-based surveys, across all three types of models, the best AICc model predictions and the model averaged predictions were very similar, so although I didn’t include model averaged predictions in my comparisons, I expect them to be very similar to the best model. I chose to use the best AICc model because it would be simplier.

Models for both types of surveys tended to provide unbiased predictions of the number of redds in the stream. Ground-based surveys required slightly less of an adjustment (in either direction) than air-based surveys, but there was certainly an adjustment in both cases, highlighting the need for such observer error models. The median absolute bias for both types of surveys and for both modeling approaches was less than one redd, compared to an undercount of 2 redds for ground-based surveys, and 6 for air-based surveys.

For the ground-based surveys, the most important covariates to explain net error were observed redd density and observer experience, whereas most of the possible covariates had similar importance for omission and commission models. However, the predictions of total redds are almost identical regardless of whether one uses the net error or omission / commission models, or the best model by AICc, or the naive model (only random effects). This suggests that the fixed effects covariates included in this study are not explaining much of the variation in error rates, but that there is information in the random effects of reach and surveyor sufficient to make unbiased predictions. However, those random effects cannot be easily carried on to another study area, or even another year, which limits their usefulness as a predictive model.

For the air-based surveys, AveSunny figured prominently in net error and commmission models, whereas observed redd density was clearly the most important for explaining omission errors. The net error and omission / commission models, either the best AICc or the naive versions, all made very similar unbiased predictions, although they appear to slightly under-predict at higher number of redds.

The fact that the predictions from the naive model (random effects only) are highly correlated with the best and model averaged versions for all the various error models, except for net errors for air surveys, suggests that the fixed effects are not explaining much of the observed variation in error rates. This is also seen in Figure 3 and Figure 13. The root mean squared error (RMSE) of predictions is slightly smaller when using the best AICc model compared to the naive model, so there is some benefit to those fixed effects.

References

Bates, D., M. Mächler, B. Bolker, and S. Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1):1–48.

Nakagawa, S., and H. Schielzeth. 2013. A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4(2):133–142.

R Core Team. 2019. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.

  1. Biometrician, Biomark, Inc., ↩

  2. Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region, ↩